The present invention relates to imaging generally and more particularly to post acquisition image processing generally.
Post acquisition image processing is well known in the literature. Publications which describe the general state of the art in post acquisition image processing are : Lim, J. S. xe2x80x9cTwo dimensional signal and image processingxe2x80x9d. Englewood Cliffs, N.J. Prentice Hall. (1990); Russ, J. C. xe2x80x9cImage processing handbookxe2x80x9d. CRC Press. (1992); Pratt, W. K. xe2x80x9cDigital image processingxe2x80x9d. NY John Wiley and Sons, Inc. (1991); Rosenfeld, A. and Kak, A. C. xe2x80x9cDigital picture processingxe2x80x9d. Academic Press. (1976); Castleman, K. R. xe2x80x9cDigital Image Processingxe2x80x9d. Prentice-Hall Inc. Englewood Cliffs, N.J. (1979).
Imaging which provides information relating to refractive characteristics in a imaged volume is known for extremely limited applications. In microscopy, Smith, Nomarski and Differential Interference Contrast (DIC) imaging is known and is described in the following publications: Nomarski, G. xe2x80x9cMicrointerferometre differential a ondes polariseesxe2x80x9d. J. Phys. Radium 16:9S-11S (1955); Lang, W. xe2x80x9cDifferential-Interferenz-Miikroskopiexe2x80x9d. Carl-Zeiss, Oberkochen (1975); Inoui, S. and Spring, K. S. xe2x80x9cVideo Microscopy: the fundamentalsxe2x80x9d. 2nd edition. Plenum Press, NY. (1997). Tanford, C. xe2x80x9cPhysical chemistry of macromoleculesxe2x80x9d. John Wiley NY. (1961). Appendix C describes classical Rayleigh interference methods, Philpot and Svenson methods based on schlieren image, Lamm method of line displacement, and Gouy interference method all developed for determination of one dimensional refractive index variations.
Computer analysis of DIC imaging is not readily achieved. Known instances are described in the following publications: Allen, R. D., Allen, N. S. and Travis, J. L. xe2x80x9cVideo-enhanced contrast, differential interference contrast (AVEC-DIC) microscopy: a new method capable of analyzing microtubule related motility in the reticulopodial network of Allogromia laticollaria.xe2x80x9d Cell Motility 1: 291-302 (1981); Cogswell, C. J. and Sheppard, C. J. R. xe2x80x9cConfocal differential interference contrast (DIC) microscopy: including a theoretical analysis of conventional and confocal DIC imagingxe2x80x9d. J. Microsc. 165:81-101 (1992); Gelles, J., Schnapp, B. J. and Sheetz, M. P. xe2x80x9cTracking kinesin-driven movements with nanometre-scale precisionxe2x80x9d, Nature 331:450-453 (1988); Hdusler, G. and Kvrner, E. xe2x80x9cImaging with expanded depth of focusxe2x80x9d. Zeiss Inform. 29: 9-13 (1987); Preza, C., Snyder, D. L. and Conchello, J-A. xe2x80x9cImage reconstruction for three-dimensional transmitted light DIC microscopyxe2x80x9d. SPIE 2984:220-231 (1997);.Schormann, T. and Jovin, T. M. xe2x80x9cContrast enhancement and depth perception in three-dimensional representations of differential interference contrast and confocal scanning laser microscope imagesxe2x80x9d. J. Microsc. 166:155-168 (1992).
Computerized ray tracing between discrete refractive and reflective surfaces is extremely well developed, but is not well known in the environment of non-homogeneous indices of refraction. This area is described in the following publications: Hecht E. and Zajac A. xe2x80x9cOpticsxe2x80x9d 2nd ed. Addison-Wesley Reading MA (1997); Jenkins, F. A, and White, H. E. xe2x80x9cFundamentals of opticsxe2x80x9d. McGraw-Hill, NY (1950) ch.8: Ray Tracing.
Calculation of point spread functions (PSF) is extremely well known as described in the following publication: Born M. and Wolf E. xe2x80x9cPrinciples of Opticsxe2x80x9d Pergamon London (1959);Goodman J. W. xe2x80x9cStatistical Opticsxe2x80x9d John Wiley and Sons NY (1985); Hecht E. and Zajac A. xe2x80x9cOpticsxe2x80x9d 2nd ed. Addison-Wesley Reading MA. (1997); Gibson S. F. and Lanni F. xe2x80x9cDiffraction by circular aperture as a model for three-dimensional optical microscopyxe2x80x9d. Opt. Soc. Am. A 6:1357-1367 (1989); Gibson S. F. and Lanni F. xe2x80x9cModeling aberrations due to mismatched layers for 3-D microscopyxe2x80x9d SPIE optics in complex systems 1319:470-471 (1990); Gibson S. F. and Lanni F. xe2x80x9cExperimental test of an analytical model of aberration in an oil-immersion objective lens used in three-dimensional light microscopyxe2x80x9d. J. Opt. Soc. Am. A 8:1601-1613 (1991).
Deconvolution of three dimensional microscopic images having location independent PSF is well known and is described in the following publications, some of them authored by some of the present inventors: Jansson, P. A. ed. xe2x80x9cDeconvolution of images and spectraxe2x80x9d. Academic Press NY (1997); Agard, D. A. and Sedat, J. W. xe2x80x9cThree-dimensional architecture of a polytene nucleusxe2x80x9d. Nature 302:676-681 (1984); Agard, D. A., Hiraoka, Y., Shaw, P. and Sedat, J. W. xe2x80x9cFluorescence microscopy in three dimensionsxe2x80x9d. Methods in Cell Biology 30: 353-377 (1989); Castleman, K. R. xe2x80x9cDigital Image Processingxe2x80x9d. Prentice-Hall Inc. Englewood Cliffs, N.J. (1979). Correction of telescopic images by the use of suitably distorted mirrors and deconvolution of two dimensional telescope images having location dependent PSF are described in the following publications: Boden, A. F., Reeding, D. C, Hanisch, R. J., Mo, J. and White, R. xe2x80x9cComparative results with massively parallel spatially-variant maximum likelihood image restorationxe2x80x9d. Bul Am Astr. Soc 27:924-929 (1995); Boden, A. F., Reeding, D. C, Hanisch, R. J. and Mo, J. xe2x80x9cMassively parallel spatially-variant maximum likelihood restoration of Hubble space telescope imageryxe2x80x9d. J Opt Soc Am A 13: 1537-1545 (1996); Jansson, P. A. ed. xe2x80x9cDeconvolution of images and spectraxe2x80x9d. Academic Press NY (1997); Tyson R. K. xe2x80x9cPrinciples of Adaptive Opticsxe2x80x9d Academic Press NY (1991). Reconstruction of blurred images from point objects is described in the following publications: Carrington, W. A., Lynch, R. M., Moore, D. W., Isenberg, G., Fogarty, K. E. and Fay, F. S. xe2x80x9cSuperresolution three-dimensional images of fluorescence in cells with minimal light exposurexe2x80x9d. Science 268:1483-1487 (1995); Femino, A. M., Fay, F. S., Fogarty, K., and Singer, R. H. xe2x80x9cVisualization of single RNA transcripts in situxe2x80x9d. Science 280:585-590 (1998).
The present invention seeks to provide improved apparatus and techniques for post acquisition image processing.
There is thus provided in accordance with a preferred embodiment of the present invention apparatus for computational adaptive imaging including an image information acquirer providing information relating to the refractive characteristics in a three-dimensional imaged volume, a ray tracer, utilizing the information relating to the refractive characteristics to trace a multiplicity of rays from a multiplicity of locations in the three-dimensional imaged volume through the three-dimensional imaged volume, thereby providing a location dependent point spread function and a deconvolver, utilizing the location dependent point spread function, to provide an output image corrected for distortions due to variations in the refractive characteristics in the three-dimensional imaged volume.
Preferably, the image information acquirer acquires at least two three-dimensional images of a three-dimensional imaged volume, at least one of the two three-dimensional images containing the information relating to the refractive characteristics in a three-dimensional imaged volume.
When the refractive characteristics are extractable from the image to be corrected for distortions, or are known independently, only one three-dimensional image need be acquired.
The acquirer may obtain refractive index information from DIC, for example from phase microscopy or from flourescence -for example in DNA associated stains wherein the stain intensity is proportional to the refractive index increment.
Refractive index mapping may be applied to samples whose refractive index is known. For example this may apply to microchip wafer structures, whose geometry is known.
In accordance with a preferred embodiment of the present invention, the image acquirer acquires at least three three-dimensional images of the three-dimensional imaged volume.
Preferably, the image acquirer acquires a plurality of three-dimensional images of the three-dimensional imaged volume, each the image having a discrete wavelength band.
Alternatively, the image acquirer acquires a multiplicity of three-dimensional images of the three-dimensional imaged volume, each the image having a wavelength band which is part of a continuum represented by the wavelength bands of the multiplicity of three-dimensional images.
In accordance with a preferred embodiment of the present invention, the ray tracer and the deconvolver utilize the information relating to the refractive characteristics in a three-dimensional imaged volume obtained from one of the three-dimensional images to correct at least another one of the three-dimensional image or itself The acquirer may obtain refractive index information from DIC, or from phase microscopy or from flourescence for example in DNA associated stains wherein the stain intensity is proportional to the refractive index increment.
Refractive index mapping may be applied to samples whose refractive index is known. For example this may apply to microchip wafer structures, whose geometry is known.
According to one embodiment of the present invention, the three-dimensional images are electromagnetic energy images. Preferably, the three-dimensional images are infrared images.
Alternatively, the three-dimensional images are non-electromagnetic images.
In accordance with a preferred embodiment of the present invention, the image acquirer receives digital image data from a digital image source and derives therefrom the information relating to the refractive characteristics in a three-dimensional imaged volume.
Preferably, the ray tracer and the deconvolver operate repeatedly over time to provide a multiplicity of output images, each corrected for distortions due to variations in the refractive characteristics in the three-dimensional imaged volume. Such a deconvolution process can be iteratively applied to the whole process to improve the estimation of the refractive index.
In accordance with one embodiment of the present invention, the output image is an acoustic image and the refractive characteristics are characteristics of a material which the passage of acoustic energy therethrough.
In accordance with an alternative embodiment of the present invention, the output image is an electromagnetic image and the refractive characteristics are characteristics of a material which the passage of electromagnetic energy therethrough.
There is also provided in accordance with a preferred embodiment of the present invention a method for computational adaptive imaging including the steps of:
providing information relating to the refractive characteristics in a three-dimensional imaged volume;
ray tracing, utilizing the information relating to the refractive characteristics, a multiplicity of rays from a multiplicity of locations in the three-dimensional imaged volume through the three-dimensional imaged volume, thereby providing a location dependent point spread function; and
deconvoluting, utilizing the location dependent point spread function, thereby providing an output image corrected for distortions due to variations in the refractive characteristics in the three-dimensional imaged volume. Such a deconvolution process can be iteratively applied to the whole process to improve the estimation of the refractive index.
Preferably, the step of providing information includes acquiring at least two three-dimensional images of a three-dimensional imaged volume, at least one of the two three-dimensional images containing the information relating to the refractive characteristics in a three-dimensional imaged volume. When the refractive characteristics are extractable from the image to be corrected for distortions, or are known independently, only one three-dimensional image need be acquired.
In accordance with a preferred embodiment of the present invention, the step of providing information includes acquiring at least three three-dimensional images of a three-dimensional imaged volume.
Preferably, the step of providing information includes acquiring a plurality of three-dimensional images of the three-dimensional imaged volume, each image having a discrete wavelength band.
In accordance with a preferred embodiment of the present invention, the step of providing information includes acquiring a multiplicity of three-dimensional images of the three-dimensional imaged volume, each the image having a wavelength band which is part of a continuum represented by the wavelength bands of the multiplicity of three-dimensional images.
In accordance with one embodiment of the present invention, the three-dimensional images are electromagnetic energy images. Preferably, the three-dimensional images are infrared images.
In accordance with an alternative embodiment of the present invention, the three-dimensional images are non-electromagnetic images.
The refractive index may be in any medium and the imaging method may be for a generalised method for inhomogeneous media that may distort the image.
Preferably, the step of providing includes receiving digital image data from a digital image source and deriving therefrom the information relating to the refractive characteristics along a multiplicity of light paths in a three-dimensional imaged volume.
In accordance with a preferred embodiment of the present invention, the steps of providing information, ray tracing and deconvoluting operate repeatedly over time to provide a multiplicity of output images, each corrected for distortions due to variations in the refractive characteristics in the three-dimensional imaged volume. Such a deconvolution process may be iteratively applied to the whole process to improve the estimation for the refractive index map.
There is also provided in accordance with a preferred embodiment of the present invention apparatus for utilizing differential interference contrast images to provide three-dimensional refractive index information including a line integrator operating on differential interference contrast images displaying a directional derivative of refractive index of an object to invert the directional derivative thereof, thereby providing a plurality of two-dimensional representations of the refractive index of the object. Alternatively the three-dimensional refractive index map can be obtained from phase microscopy, or from fluorescence where the staining is proportional to the refractive index increment.
There is additionally provided in accordance with a preferred embodiment of the present invention apparatus for utilizing differential interference contrast images to provide three-dimensional refractive index information and also including a deconvolver performing deconvolution of the plurality of two-dimensional representations of the refractive index of the object, thereby reducing out-of-focus contributions to the two-dimensional representations of the refractive index of the object.
There is further provided in accordance with a preferred embodiment of the present invention a method for utilizing differential interference contrast images to provide three-dimensional refractive index information including performing line integration on differential interference contrast images displaying a directional derivative of refractive index of an object to invert the directional derivative thereof, thereby providing a plurality of two-dimensional representations of the refractive index of the object. Alternatively the three-dimensional refractive index map can be obtained from phase microscopy, or from flourexcence where the staining is proportional to the refractive index increment.
Additionally in accordance with a preferred embodiment of the present invention there is provided a method for utilizing differential interference contrast images to provide three-dimensional refractive index information and also including performing deconvolution of the plurality of two-dimensional representations of the refractive index of the object, thereby reducing out-of-focus contributions to the two-dimensional representations of the refractive index of the object. Again, the three-dimensional refractive index map can be obtained from phase microscopy, or from flourexcence where the staining is proportional to the refractive index increment.
Further in accordance with a preferred embodiment of the present invention there is provided apparatus for ray tracing through a medium having multiple variations in refractive index including:
a computer employing an analytically determined path of a ray through the multiplicity of locations in the medium, for a plurality of rays impinging thereon in different directions, by utilizing known local variation of the refractive index at a multiplicity of locations in the medium.
There is additionally provided in accordance with a preferred embodiment of the present invention a method of ray tracing through a medium having multiple variations in refractive index including:
determining local variation of the refractive index at a multiplicity of locations in the medium;
analytically determining the path of a ray through the multiplicity of locations in the medium, for a plurality of rays impinging thereon in different directions. The ray tracing may also include the computation of absorptions, reflections and scattering of rays and their contributions to the imaging process.
Still further in accordance with a preferred embodiment of the present invention there is provided apparatus for confocal microscopy including:
a ray tracer, employing known variations of the refractive index in a three-dimensional sample for determining the paths of a multiplicity of rays emerging from at least one point in the sample and passing through the sample, thereby determining an aberrated wavefront for each the point; and
an adaptive optics controller utilizing the aberrated wavefront to control an adaptive optical element in a confocal microscope, thereby to correct aberrations resulting from the variations in the refractive index.
There is additionally provided in accordance with a preferred embodiment of the present invention a method for confocal microscopy including:
determining variations of the refractive index in a three-dimensional sample;
determining the paths of a multiplicity of rays emerging from at least one point in the sample and passing through the sample, thereby determining an aberrated wavefront for each the point; and
utilizing the aberrated wavefront to control an adaptive optical element in a confocal microscope, thereby to correct aberrationresulting from the variations in the refractive index.
There is additionally provided in accordance with a preferred embodiment of the present invention a method for adding (computationally or physically) in the imaging path a three-dimensional medium (anti-sample) with refractive properties that correct for the distortions of the three-dimensional sample.